Mapping out the multifunctionality of speakers’ gestures

Although it is widely acknowledged that gestures are complex functional elements of human communication, many current functional classification systems are rather rigid, implicitly assuming gestures to perform only one function at any given time. In this paper, we present a theoretical view on the inherent multifunctionality of speakers’ gestures, inspired by frameworks in structural-functional linguistics (Halliday’s Systemic Functional Grammar and Hengeveld & Mackenzie’s Functional Discourse Grammar). Building upon this view, we report on a large-scale internet-based gesture perception study, designed in a way open to the potential for complex multifunctionality of gestural expression. The results provide quantitative support for the view that speakers’ gestures typically contribute to multiple semantic and meta-communicative functions of the ongoing discourse simultaneously. Furthermore, we identify clusters of functions that tend to be combined, as well as correlations between pairs of them. As a whole, this paper achieves some degree of convergence between ecological and experimental views on gesture functionality. Kasper Kok, is a PhD candidate at VU University, Amsterdam. His research focuses on the incorporation of speaker’s gestures in cognitive and functional models of grammar. Dr. Kirsten Bergmann is a post-doctoral researcher at Bielefeld University. Her research is on speech and gesture use in human-humanand human-computer interaction. Prof. Alan Cienki is professor of language use and cognition at VU University, Amsterdam, and is director of the Multimodal Cognition and Communication Laboratory (PoliMod) at Moscow State Linguistic University. Prof. Stefan Kopp is the head of the Social Cognitive Systems Group at Bielefeld University.